Intransitivity in Coevolution

We review and investigate the current status of intransitivity as a potential obstacle in coevolution. Pareto-Coevolution avoids intransitivity by translating any standard superiority relation into a transitive Pareto-dominance relation. Even for transitive problems though, cycling is possible. Recently however, algorithms that provide monotonic progress for Pareto-Coevolution have become available. The use of such algorithms avoids cycling, whether caused by intransitivity or not. We investigate this in experiments with two intransitive test problems, and find that the IPCA and LAPCA archive methods establish monotonic progress on both test problems, thereby substantially outperforming the same method without an archive. Coevolution offers algorithms for problems where the performance of individuals can be evaluated using tests [1–7]. Since evaluation in coevolution is based on evolving individuals, coevolution setups can suffer from inaccurate evaluation, leading to problems such as over-specialization, Red Queen dynamics, and disengagement [8, 9]. A problem feature that has received particular interest in the past is that of intransitivity [9]. A relation R is transitive if aRb ∧ bRc implies aRc; if this cannot be guaranteed, the relation is intransitive. An example of a problem where the relation used to compare individuals is intransitive, is Rock, Paper, Scissors; while scissors beats paper and paper beats rock, scissors is beaten by rock. The existence of such intransitive relations in a coevolution problem can lead to cycling, i.e. the recurrence of previously visited states of the population. Intransitivity has been viewed as an inherent feature of coevolution that can render algorithms unreliable. Indeed, the resulting problem of cycling has been thought of as an obstacle that could prevent coevolution from becoming a reliable problem solving technique, as attested to by the following quote: “We believe that the cycling problem, like the local minima problem in gradient-descent methods..., is an intrinsic problem of coevolution that cannot be eliminated completely” [10]. Recently, it has been shown that coevolution can in principle approximate ideal evaluation [11], i.e. equivalence to evaluation on the space of all tests. This result is based on the solution concept offered by Pareto-Coevolution [12, 13], consisting of all candidate solutions whose performance cannot be improved on any test without decreasing the individual’s outcome on some other test. Another approach to achieve reliability in coevolution is to use an archive to maintain promising candidate solutions and informative tests. If an archive can avoid regress, then generating all possible individuals with non-zero probability guarantees that the algorithm can only make progress and will occasionally do so, thus enabling the coevolutionary goal of open-ended, sustained progress. Rosin’s covering competitive algorithm [14], alternates between finding a firstplayer strategy that beats all second-player strategies in the archive and vice versa. This guarantees that regress can be avoided, but the strict criterion of defeating all previous opposition is likely to result in stalling as soon as mutually exclusive tests exist, i.e. tests that cannot all be solved by a single learner but can be solved individually by different learners. Ficici and Pollack’s Nash Memory [15] guarantees progress for the solution concept of the Nash Equilibrium. It is limited to symmetric games, but extension to the case of asymmetric games is noted to be straightforward. The Incremental Pareto-Coevolution Archive (IPCA) [16] guarantees progress for Pareto-Coevolution, and is applicable to both symmetric and asymmetric problems. All of the above archive methods can only guarantee progress however if the archive size is unlimited. A layered variant of IPCA was found empirically to produce reliable progress on test problems using a bounded archive [17]. Our aim here is to investigate the role of intransitivity in coevolution in the light of the above developments. It is known that the use of Pareto-coevolution transforms intransitive superiority relations into transitive relations; we provide a concise proof demonstrating this. While this transitive relation provides an appropriate basis for reliable coevolution, cycling is also possible for transitive relations, as shown in a simple example. We discuss the potential of coevolution methods to avoid cycling, and investigate the performance of two recent algorithms on test problems featuring intransitivity. The structure of this paper is as follows. Section 1 discusses intransitivity and recent results in achieving reliable progress in coevolution. Section 2 presents experimental results, and Section 3 concludes.